Scalable and dynamic quality of service control

ABSTRACT

Applications and users dynamically make QoS provisioning requests for individual traffic flows traversing client and server hosts. A traffic flow provisioning request is conveyed to a services manager, which determines a set of traffic attributes for the flow and determines the networks the flow traverses between the client and server hosts. The services manger then oversees the admission of the flow to appropriate traffic classes in each determined network and the obtaining of a DSCP value for each network. Lastly, the services manger conveys the DSCP value of the first network traversed back to the client or server host, depending on the direction of the flow, which host is then configured to appropriately mark the DSCP field of the traffic flow packets. In a further embodiment, the services manager also instructs the client or server host to perform packet policing and shaping for the flow.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a Continuation of U.S. application Ser. No.10/706,796, filed Nov. 12, 2003, incorporated herein by reference in itsentirety.

STATEMENT OF GOVERNMENT SUPPORT

This invention was made with Government support under agreement No.F30602-00-C-0009 awarded by the Air Force Research Laboratory andagreement Nos. DASG60-01-C-0058 and DASG60-00-C-0060 awarded by the U.S.Army Space and Strategic Defense Command. The Government has certainrights.

FIELD

The disclosure relates generally to application and user control overnetwork quality of service for application sessions. More particularly,the disclosure relates to methods and systems for scalable, dynamic,end-to-end, quality of service control within a data network on a perapplication traffic flow basis.

DESCRIPTION OF THE BACKGROUND

Users have an increasing need to dynamically prioritize the individualtraffic flows of application sessions traversing IP (Internet Protocol)based networks, such as enterprise networks and VPN-type servicesprovided by Internet Service Providers. An application session, for thepurposes of this discussion, is a connection between any given instanceof a client application and a server application, wherein the sessioncomprises one or more individual traffic flows. The need to prioritizetraffic flows exists because networks are carrying increasingly diversetraffic types that all compete for network resources. When a giventraffic flow has particular bandwidth, latency, and packet lossrequirements (i.e., particular quality of service (QoS) needs), othernetwork traffic can consume network resources and thereby adverselyaffect the flow. Network administrators will typically use Diff Serv(differentiated services) enabled networks to allocate network resourcesto critical applications so that the performance needs of theseapplication sessions are met.

Specifically, a Diff Serv enabled network allows heterogeneous traffictypes from different application sessions to co-exist while at the sametime allocating these varying traffic types different network resourcesbased on the needs of the application. As indicated, some traffic typeshave bandwidth, latency, and packet loss requirements that the total mixof traffic on a network can adversely affect. Diff Serv allows a networkadministrator to establish traffic classes in the network by allocatingnetwork resources to each class. (In general, traffic classes areestablished by configuring the network routers to support varying packetqueuing and forwarding techniques and thereby providing differentthroughput, latency, and loss services for each class). Applicationsessions are then assigned to these classes and are thereby generallyable to receive a certain quality of service.

Traffic classes are not specific to any given user, application, orsession. Rather, any given traffic class will typically carry datapackets from many different applications originating from many differentusers. In particular, edge routers of the network (or a dedicatedhardware coinciding with the router) assign packets from applicationsessions to traffic classes. The edge routers analyze each packet as itenters the network, classifying the packet according to an assignedtraffic class, and insert a specific Differentiated Services Code Point(DSCP) value in the DSCP field of the packet based on the assignedtraffic class. The network routers use a packet's DSCP value to processthe packet according to the assigned traffic class. The edge routersalso typically condition the traffic before forwarding it into thenetwork. Traffic conditioning includes packet shaping, dropping, andpolicing. Policing ensures that any given application session does notutilize more bandwidth than associated with the assigned service.Importantly, for edge router performance reasons, the networkadministrator typically configures the edge routers to assign trafficflows to traffic classes on an application and site basis. In otherwords, all traffic flows from a given application session from a set ofdesignated sites are assigned to the same traffic class.

Although the traditional approaches to Diff Serv can provide criticalapplications with the required resources to the meet the applicationneeds, these approaches have several drawbacks. In particular, there arean increasing number of application sessions that users want toprioritize. At the same time, however, very often only a particulartraffic flow from a given session requires prioritization. Similarly,users often want to dynamically decide when a given application sessionreceives prioritized service and when it does not. However, typical DiffServ approaches are not sufficiently scalable and dynamicallyconfigurable to meet the increasing number of applications sessions andindividual user needs. Another issue is that Diff Serv approaches areoften susceptible to unpredictable traffic loads.

More specifically, the assignment of application sessions to trafficclasses is static under Diff Serv. Once a network administratorconfigures the edge routers to mark specific application packets, allapplication sessions for that application from a set of designated sitesare assigned to the given traffic class and these configurations areessentially static. This is problematic because in any given instance, auser may decide that a given application session requires the servicesof a particular traffic class and the network administrator has notconfigured the edge routers to assign that user to the class or,similarly, may not have assigned that given application to a trafficclass. In addition, users often determine that a given session does notrequire the services of a given traffic class or requires a higherservice. One solution to these issues is for a user to dynamicallycontact the network administrator for reconfiguration of the edgerouters on a per need basis. However, this is both time-consuming andunusable from a user perspective and unfeasible from an administrativeperspective.

A similar issue is that the network resources are limited. Accordingly,administrators must take care when configuring the edge routers to notover admit application sessions/users to the traffic classes. However,not all admitted users simultaneously run application sessions, whichresults in administrators essentially underestimating the number ofapplication sessions that should be allowed access to a traffic classand thereby under-utilizing the network resources. Similarly, it isusually only a given traffic flow within a given session that actuallyrequires the services, causing traffic classes to often carryunnecessary data packets. Accordingly, administrators should allocatenetwork resources on a per-traffic flow basis and when specificallyneeded. Again, such a system is time consuming and unusable from a userperspective and unfeasible from an administrative perspective. Inaddition, such continuous/dynamic assignment of traffic flows to trafficclasses is not scalable, which is further described below.

Similar issues exist in network environments where IP addresses aredynamically assigned through DHCP (Dynamic Host Configuration Protocol)and therefore dynamically change. In this environment, networkadministrators are required to constantly reconfigure the edge routersas IP addresses are re-assigned to network users or are forced to assignapplications to traffic classes at a more coarse level, disregarding theindividual sites. Again, quality of service is more appropriatelydesignated on a per user/application session level.

A second issue with traditional Diff Serv approaches is that they arenot scalable. On a fundamental basis, as more users are added to anetwork and/or as administrators increase the number of applicationsrequesting traffic class services, the edge routers correspondingly mustperform additional packet filtering/marking/conditioning/policingresponsibilities. At some point, the functional ability of the edgerouters is reached, thereby causing the administrators to have to addadditional edge routers to the network at additional cost. Similarly, asthe edge routers are configured to admit more application sessions, theinternal network resources must also be increased. However, as describedabove, the number of application sessions simultaneously needing thenetwork resources is usually less than the number the edge routers areconfigured to admit, thereby causing network administrators to overprovision the network.

Similarly, as described above, quality of service is more appropriatelydesignated on a dynamic user basis at the traffic flow level. Again,edge router performance issues will occur if routers are made to markand condition at the traffic flow level, especially as users andapplications are added to the network. In addition, continuouslyconfiguring and reconfiguring edge routers on a per user basiscompromises edge router performance.

A third issue with the traditional Diff Serv approaches is admissioncontrol. Typically, when configuring network resources to establishtraffic classes, an administrator approximates the number ofinstantaneous application sessions that will occur and the trafficclasses each should be assigned to. As long as the number ofsimultaneous application sessions for a given traffic class does notexceed the configured approximation, each session will receive adequateresources. However, an unexpected number of sessions for a given trafficclass can occur, which results in performance degradation across theclass causing no session to receive adequate resources. One solution tothis is to over-provision the traffic class, but this wastes networkresources.

Prior solutions have only partially addressed the above issues,including systems referred to as bandwidth broker systems. A bandwidthbroker is a network management system that provides admission controland bandwidth management for Diff Serv enabled networks by tracking theamount of bandwidth being used in each traffic class. When anapplication session requires the services of a traffic class, theapplication makes an admission request to the bandwidth broker,providing the bandwidth and the traffic class being requested. Thebandwidth broker determines whether sufficient bandwidth is available inthe respective traffic class. If there is sufficient bandwidth, thebandwidth broker decreases the amount of available bandwidth for thetraffic class, and configures the appropriate edge router to mark,condition, and police the new traffic. Otherwise the new session isdenied admission.

Advantageously, bandwidth broker systems prevent traffic classes frombecoming over-utilized thereby preventing performance degradation; theyallow for dynamic/automatic application controlled QoS management; and,they allow for better utilization of internal network resources.However, the systems have several disadvantages. First, bandwidth brokersystems fail to address the scalability issues described above. Whilethese systems are suitable for small-scale networks, edge routers arenot capable, from a performance perspective, ofmarking/conditioning/policing numerous application sessions or worse,traffic flows, and are compromised as users increasingly make numerousconfigurations/reconfigurations on a dynamic basis. A second issue isthat applications must be specifically designed/programmed tocommunicate with bandwidth broker systems and, as such, applications arenot easily integrated into the framework making bandwidth broker systemsdifficult to use. More specifically, an application developer mustunderstand the low-level detailed traffic attributes of the applicationsessions the application will generate and translate these attributesinto one of the traffic classes supported by the network so that theactual traffic class request can be made to the bandwidth broker. Thisissue becomes worse if an application session needs to traverse severalnetworks, each with different traffic classes and managed by differentbandwidth brokers. Here, the application developer must make thetraffic-attributes to traffic-class translation for eachnetwork/bandwidth broker. In addition, the developer must determinewhich networks a given session will traverse in order to communicatewith the appropriate bandwidth brokers.

SUMMARY

Accordingly, it is desirable to provide methods and system that overcomethe shortcomings of the prior art and allow applications and/or users ofa network to dynamically provision QoS services on an individual trafficflow basis. Representative embodiments apply to an environment where aplurality of networks interconnect client and server computers/hostswherein each network is either a Diff Serv enabled network withpre-provisioned traffic classes or an over provisioned/congestion-freenetwork. Each client and server computer comprises one or moreclient/server applications wherein any given instance of a clientapplication establishes a session via the networks to a serverapplication and in particular, wherein the client and/or serverapplications and users of these applications desire QoS services fromthe networks for the traffic flows moving between the clients andservers.

A first embodiment comprises a middleware module at each of a pluralityof client and/or server hosts, a services control module, and one ormore network control systems. Client and server applications directlyinterface the middleware module in order to make QoS requests.Specifically, once a client application and a server applicationestablish a session, the client application, and similarly the serverapplication, dynamically and automatically determines when one the giventraffic flows it is generating needs QoS services from the networks andmakes a QoS request to the middleware module specifying a flowidentification. The middleware module conveys this request to theservices module, which determines low-level traffic flow attributes forthe flow based on the flow identification and determines the networksthe flow traverses between the client and server hosts. Based on thedetermined networks, the services manager then oversees admissioncontrol of the traffic flow to these networks.

Specifically, for each Diff Serv enabled network, either a networkcontrol system or the services manager itself performs QoS admissioncontrol and bandwidth management for that network. In particular, when anetwork control system is associated with a network, it performsadmission control for new traffic flows to the pre-provisioned trafficclasses provisioned in that network. For an admitted traffic flow to aspecified traffic class, the network control system returns a DSCP valuecorresponding to the traffic class. Similarly, the clients and serversmay already have an explicit service level agreement with the providerof a network. Here, the services manager directly performs admissioncontrol over the service level agreement.

Accordingly, based on a QOS request for a given traffic flow between aclient and server, the services manager maps the determined traffic flowattributes to one of the traffic classes supported by each of thedetermined Diff Serv enabled networks and oversees admission control tothese networks by interfacing with the network control systems and/ordirectly. Assuming each network control system and/or the servicemanager can admit the new traffic flow to a traffic class of itscorresponding network, each determines a DSCP value for its trafficclass and returns this DSCP value to the services manager. The servicesmanager then returns to the middleware module at the client or server,depending on the source of the traffic flow, the DSCP valuecorresponding to the first network the flow traverses. Upon receivingthe DSCP value from the services manager, the middleware moduleconfigures the client/server to appropriately mark the DSCP field of thetransmitted traffic flow packets thereby completing the overall QoSconfiguration.

In accordance with a second embodiment, client and server applicationscontinue to make QoS provisioning requests for the traffic flows eachdirectly generates. However, each application can also make QoSprovisioning requests on behalf of the other.

For example, a client application can make QoS requests for the trafficflows it generates and for the traffic flows a corresponding serverapplication generates. Similar to the first embodiment, this embodimentcomprises a middleware module at client and/or server hosts, a servicesmanager, and network control systems, all of which continue to functionas with the first embodiment. In addition, this embodiment alsocomprises a policy enforcement module at each of a plurality of theclient and/or server hosts. When resident at a given host, this moduleperforms DSCP marking configurations for the traffic flows theapplications at this given host generates but wherein the module isactivated by an opposing host.

As an example of this second embodiment, a client can comprise amiddleware module and a server can comprise a policy enforcement module.Assume applications at the client and server have an application sessionwith client-to-server and server-to-client flows. The client applicationcan make a QoS provisioning request for the client-to-server flow, withthe middleware module relaying the request to the services manager andwith the middleware module also configuring the client with the obtainedDSCP value for packet marking. In addition, the client application canalso make QoS provisioning requests for the server-to-client flow.Again, the middleware module relays the request to the services managerand the services manager determines traffic attributes for the flow,determines the networks the traffic flow traverses, and overseesadmission control obtaining a DSCP value. However, rather than conveyingthe DSCP value back to the middleware module at the client, the servicesmanager now conveys the DSCP value to the policy enforcement module atthe server. The policy enforcement module, upon receiving the DSCP valuefrom the services manager, configures the server to intercept thetraffic flow packets generated by the server application prior totransmission and to appropriately mark the DSCP field of the packets,completing the overall QoS configuration.

In accordance with a third embodiment, in addition to performing thepacket marking function, the client and server hosts can also performpacket policing and packet shaping functions for the traffic flowscorresponding applications generate. Similar to the first embodiment,this embodiment comprises a middleware module at the client and/orserver hosts, a services manager, and network control systems, each ofwhich continue to function the same. In addition, similar to the secondembodiment, this third embodiment can also comprise a policy enforcementmodule at remote hosts, which again, applications at a local host use tomake QoS requests for the traffic flows a remote host generates.However, in accordance with this third embodiment, this policyenforcement module, under the instruction of the services manager, alsoperforms packet policing and packet shaping functions for the trafficflows applications at the remote host generate. In addition, in the casewhere an application at a local host is making QoS requests for thetraffic flows it directly generates, this third embodiment adds a policyenforcement module to the local host that performs the packet policingand packet shaping functions for these traffic flows. In particular,under the control of the services manager, the middleware module at thelocal host continues to perform the packet marking function. However,the policy enforcement module performs the packet policing and shapingfunctions with respect to these flows.

In accordance with a fourth embodiment, applications can continue todirectly control the QoS requests for a given applicationsession/traffic flow; but, in addition, a user can also make QoSrequests for application sessions/traffic flows, in particular, when theapplication itself is not able to interact directly with the middlewaremodule. This embodiment again comprises a services manager and one ormore network control systems that oversee the QoS management for thenetworks on behalf of the clients and servers. In addition, thisembodiment comprises a signaling client and a middleware module at eachof a plurality of the client and/or server hosts that operate togetherto allow users to make the QoS requests for client-to-server andserver-to-client flows. Lastly, this embodiment comprises the policyenforcement module at the client and/or server hosts, which performs thepacket marking (and policing/shaping functions) whenever the signalingclient is used to initiate QoS requests.

Assuming the signaling client/middleware module are located at theclient side of an application communication, a user of the client canmake QoS requests for the client-to-server and server-to-client flowsoccurring between the client application and the corresponding serverapplication. When a user makes a QoS request for a particularapplication session/traffic flow, the signaling client conveys therequest to the services manager via the middleware module. If the flowis admitted, the services manager obtains a DSCP value and determinesthe direction of the flow. For server-to-client flows, the servicesmanager conveys the DSCP value to a policy enforcement module at theserver, which operates as above. For client-to-server traffic flows, theservices manger conveys the DSCP value to a policy enforcement module atthe client, which module then operates like the policy enforcementmodule described in reference to the second and third embodiments,configuring the client to mark the packets and possibly perform ratecontrol functions for the client-to-server traffic flows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of a first embodiment for provisioningQoS services on a per traffic flow basis, wherein the embodimentcomprises a middleware module, a services manager, and network controlsystems and wherein client and server applications communicate with themiddleware module to directly make the provisioning requests.

FIG. 2 is a functional drawing of the middleware module as shown in FIG.1.

FIG. 3 is a functional drawing of the services manager as shown in FIG.1.

FIG. 4 depicts a block diagram of a second embodiment for provisioningQoS services on a per traffic flow basis, wherein the embodimentadditionally comprises a policy enforcement module that allows a localapplication to make QoS provisioning requests for the traffic flowsgenerated by a remote application.

FIG. 5 is a functional drawing of the policy enforcement module as shownin FIG. 1.

FIG. 6 depicts a functional diagram of a third embodiment forprovisioning QoS services on a per traffic flow basis wherein inaddition to packet marking functions as in the first and secondembodiments, packet policing and packet shaping functions are alsoconducted at client and server hosts.

FIG. 7 depicts a functional diagram of a fourth embodiment forprovisioning QoS services on a per traffic flow basis, wherein userscommunicate with a signaling client to make the provisioning requests onbehalf of the client and server applications.

DETAILED DESCRIPTION

FIG. 1 is a high-level illustrative diagram of a first embodiment ofsystem 100 that provisions end-to-end QoS services on a per applicationtraffic flow basis. Specifically, system 100 allows an application todynamically, automatically, and flexibly request that the trafficflow(s) of a given session generated by the application be assigned QoSservices and then provisions the assignment, freeing the applicationfrom the details of this provisioning. System 100 comprises a servicesmanager 120, a middleware module 122 a and 122 b at client computers(102) and server computers (104) respectively, and one or more networkcontrol systems 124 a-c.

As FIG. 1 shows, system 100 applies to an environment comprising aplurality of client computers 102 a-b and server computers 104 a-binterconnected by a network 106 through a set of edge routers 108 a-b.(Note that the client and server computers are hereinafter genericallyreferred to as hosts.) Each client and server site comprises one or moreclient/server applications 110 a-b/112 a-b wherein any given instance ofa client application establishes a connection/session via network 106 toa server application and in particular, wherein the client or serverapplication desires QoS services from network 106 for the traffic flowsmoving between the client and server.

Network 106 may reside within a single administrative domain or maycomprise a plurality of interconnected sub-networks 106 a-106 e as shownin FIG. 1, with each sub-network under separate administrative control.Each sub-network 106 a-106 e is presumed to be either a Diff Servenabled network, such as sub-networks 106 a-d, with statically definedpre-provisioned traffic classes (i.e., the actual provisioning of thetraffic classes is not particular to the representative embodiment) oran over provisioned/congestion-free network, such as sub-network 106 e,with no provisioned traffic classes. While system 100 does not need toconfigure congestion free networks when provisioning an end-to-end QoSservice for a given traffic flow, system 100 needs to be aware of thesesub-networks because the given traffic flow may traverse these networks.Note also that for the Diff Serv enabled networks, each network maysupport different traffic classes.

Importantly, system 100 manages QoS services at the traffic flow levelfor any given application session between any given client and server.Hence, each time a client and a server application establish a session,system 100 allows the client and the server to control the QoS servicesfor the traffic flows occurring over that session. Specifically,applications interact with system 100 in several ways depending on thedirection of the traffic flow that requires QoS services and based onthe type of traffic flow. For example, when an instance of a clientapplication establishes a session to a server application, the clientapplication can request QoS services for the traffic flows moving fromthe client to the server. Similarly, the server application can requestQoS services for the traffic flows moving from the server to the client.In addition, the client and server applications can request QoS servicesfor the life of the application session, such that all client-to-serverand server-to-client traffic flows are mapped to a traffic class, or theclient and server can request short-lived QoS support for short-livedtraffic flows, such as bursty traffic. In this latter case, system 100enables QoS services only for the life of the bursty traffic.Accordingly, a given application instance may make numerous QoS requestsover the life of a session. Note that because system 100 allowsapplications to dynamically request QoS services on a persession/traffic flow basis when needed, network resources are moreeffectively shared/multiplexed between numerous applications overcomingnetwork scalability issues.

In general and in accordance with this first embodiment, an instance ofa client application dynamically and automatically determines when oneof the given traffic flows it is generating to an instance of a serverapplication requires QoS services from network 106 and makes a QoSrequest to system 100. Similarly, the server application instance alsodynamically and automatically determines when one of the given trafficflows it is generating to the client application instance requires QoSservices from network 106 and makes a QoS request to system 100. Uponreceiving a given request for a given traffic flow, system 100determines an appropriate traffic class supported by network 106 anddetermines if the traffic class has sufficient resources to support thetraffic flow. If not, the QoS request is rejected. However, if there aresufficient resources, system 100 provides the client or server(depending on which host made the request) with a DSCP valuecorresponding to the assigned class. The client or server then proceedsto mark the packets of the traffic flow with the DSCP value andtransmits the packets to network 106, where the network routers processthe packets according to the corresponding traffic class associated withthe DSCP value.

More specifically, once a client application 110 and server application112 establish a session, the client and/or server application conveysits QoS requirements in the form of a QoS request to the servicesmanager 120 via the middleware module 122 a/b, which is an intermediarybetween an application instance and the services manager. The servicesmanager 120 is a central entity that oversees QoS management on behalfof a group of clients 102 and servers 104 communicating over network106. Upon receiving a QoS request from an application via a middlewaremodule, the services manager translates the QoS request to a set ofdetailed traffic flow attributes, determines whether the requestcorresponds to a client-to-server flow or a server-to-client flow, anddetermines the sub-networks 106 a-e the traffic flow traverses betweenthe client and server. Based on the determined sub-networks, theservices manager then contacts the appropriate network control systems124 a-c for admission control, or indirectly handles the admissioncontrol itself.

Specifically, each sub-network 106 a-106 e is presumed to be either aDiff Serv enabled network (106 a-d) or an overprovisioned/congestion-free network (106 e). Traffic classes are notneeded for the “congestion-free” networks and as such, admission controlis not necessary for these networks. However, for each Diff Serv enablednetwork, either a network control system 124 a-c or the services manager120 performs QoS admission control and bandwidth management for thatnetwork. In particular, a network control system 124 a-c may beassociated with a sub-network (e.g., sub-networks 106 a-c) and performsadmission control for new traffic flows to the set of pre-provisionedtraffic classes provisioned in the network. The control system takes asan input a specific traffic class configured in the network and abandwidth requirement for a given traffic flow, determines if thetraffic class has sufficient available bandwidth in light of priorlyadmitted traffic flows to support the new flow, and admits or denies thenew flow based on the available bandwidth. For admitted traffic flows,the network control system returns to the services manager a DSCP valuecorresponding to the specified traffic class.

Similarly, the clients 102 and servers 104 may already have an explicitservice level agreement (comprising one or more traffic classes) withthe provider of a sub-network (such as network 106 d). Here, theservices manager performs admission control over the service levelagreement on behalf of the clients 102 and server 104 to ensure theseclients/servers do not attempt to exceed the agreement and degradeservice amongst each other. Similar to a network configuration system,the services manager takes as input an admission request to a trafficclass within the service level agreement wherein the request is for agiven traffic flow with a specific bandwidth requirement. It thendetermines if the traffic class has sufficient available bandwidth inlight of priorly admitted traffic flows to support the new flow. Ifthere is sufficient bandwidth, the services manager returns a DSCP valueassociated with the traffic class.

Accordingly, based on a QOS request for a given traffic flow between aclient and server, the services manager determines the sub-networks theflow traverses. For each Diff Serv enabled sub-network, the servicesmanager maps the determined traffic flow attributes to one of thetraffic classes supported by that sub-network. It then makes a QoSadmission request to the appropriate network control systems 124 a-cand/or makes the admission determination itself as just described.Assuming each sub-network has sufficient bandwidth to admit the newtraffic flow, each network control system (and/or the service manager)determines a DSCP value for the traffic class of its correspondingnetwork. When a client application makes the QoS request for aclient-to-server traffic flow, the services manager then returns to themiddleware module 122 a the DSCP value corresponding to the first DiffServ enabled sub-network the flow traverses. Similarly, when a serverapplication makes the QoS requests for a server-to-client traffic flow,the services manager conveys to the middleware module 122 b the DSCPvalue corresponding to the first Diff Serv enabled sub-network the flowtraverses. The services manager 120 then conveys the remaining DSCPvalues corresponding to the remaining sub-networks to the networkcontrol systems, which use these values to configure their networks toperform packet remarking (i.e., swapping of DSCP values) at the networkboundaries. Alternatively, the sub-networks can be pre-configured toautomatically perform packet remarking between corresponding trafficclasses of adjacent networks.

Upon receiving the DSCP value from the services manager for aclient-to-server flow, the middleware module 122 a configures the client102 to appropriately mark the DSCP field of the transmitted traffic flowpackets for the application session, thereby completing the overall QoSconfiguration. Similarly, for server-to-client traffic flows, themiddleware module 122 b, upon receiving the DSCP value from the servicesmanager, configures the server 104 to appropriately mark the DSCP fieldof the transmitted traffic flow packets, again completing the overallQoS configuration.

As compared to prior systems, our inventive system 100 has severaladvantages. First, system 100 allows each instance of an application todynamically make specific QOS requests as needed at the traffic flowlevel, over-coming the static configuration issues displayed by priorsystems and better utilizing network resources. Second, dynamic trafficflow level QOS management overcomes the scalability issues of priorsystems. Similarly, the scalability issues are overcome by performingthe packet-marking functions at the clients and servers. Third, whileour inventive system provides QOS admission similar to prior systems,our system further allows applications to specify QOS requirements at ahigh level, masking the applications from the low-level detailed trafficattributes. Similarly, our system automatically manages QOSconfiguration across multiple networks on behalf of applications.Reference will now be made in greater detail to the elements of therepresentative embodiment, beginning with the network configurationsystems 124, the middleware module 122, and then the services manager120.

As indicated, QoS admission control and bandwidth management for eachDiff Serv enabled sub-network within network 106 is provided through acorresponding network configuration system 124 a-c and/or by theservices manager through one or more modules we refer to as the “SLAcontrol modules” (see element 320 of FIG. 3). Each network configurationsystem 124 a-c and the SLA control modules in system 100 operatesindependently of the others and each sub-network may therefore havedifferent traffic classes provisioned and may associate different DSCPvalues with each class.

Beginning with the network configuration systems, each is similar to abandwidth broker system and as such, is aware of the traffic classesprovisioned in its corresponding sub-network, is aware of the DSCPvalues associated with each class, and is aware of the bandwidthallocated to each class. Accordingly, each network configuration systemcan be a modified version of the Telcordia Bandwidth Broker or amodified version of other known bandwidth broker systems. Commonlyassigned U.S. patent application Ser. No. 09/829,385, “Methodology forQuality of Service Provisioning for Virtual Private Networks,” which ishereby incorporated by reference, describes the Telcordia BandwidthBroker. When the network configuration system is based on the TelcordiaBandwidth Broker, the system is software-based and can reside as aseparate software process on the services manager platform 120 or as astandalone platform separate from the services manager (as shown inFIG. 1) and interconnected to the services manager through a datacommunications interface, such as interfaces 128 a-c.

Note that unlike traditional bandwidth broker systems, the networkconfiguration systems receive admission requests from the servicesmanager 120 rather than the individual applications 110 and 112. Theservices manager, as described below, prevents application developersfrom having to determine which network configuration system isresponsible for a given sub-network, from having to interface multiplesystems when an application session traverses multiple sub-networks, andfrom having to map low-level traffic attributes to one of the trafficclasses supported by a given sub-network.

The interface to any given network configuration system is a QoSadmission request to admit a given traffic flow across the configurationsystem's corresponding sub-network. Specifically, for each networkconfiguration system, the request comprises a specific traffic classsupported by the corresponding sub-network, the bandwidth requirementsof the traffic flow, and the IP address of the client and server betweenwhich the traffic flow is traversing. Based on the admission request,the network configuration system determines if the requested trafficclass has sufficient bandwidth to support the traffic flow. If thedetermined class does not have sufficient bandwidth, the networkconfiguration system informs the services manager 120 that the admissionrequest is denied. If the determined class has sufficient bandwidth, thenetwork configuration system admits the request by reducing theavailable bandwidth for that class by the amount requested. However,rather than configuring the edge routers 108 a/b to mark the new flow,the network configuration system returns the DSCP value associated withthe traffic class to the services manager. (Note that the edge routersmay continue to perform policing and possibly shaping, here on a pertraffic class basis. The network administrator would pre-provision theedge routers to perform this function.)

The SLA control module 320 of FIG. 3 is a component of the servicesmanager, but is discussed now given its functional similarity to thenetwork configuration systems. As indicated, the clients 102 and servers104 may have service level agreements (comprising one or more trafficclasses) with the providers of some sub-networks. Each of thesesub-networks has a corresponding SLA control module at the servicesmanager. Each module is software based and similar to a networkconfiguration system in that it performs admission control over thetraffic classes comprising the service level agreement for thesub-network. However, unlike a network configuration system, a SLAcontrol module does not directly communicate with its correspondingsub-network.

Each SLA control module is aware of the traffic classes comprising theservice level agreement of its corresponding sub-network, is aware ofthe DSCP value associated with each class, and is aware of the bandwidthallocated to each class. The request to a module comprises a specifictraffic class of the service level agreement and the bandwidthrequirements of the traffic flow. Based on the admission request, a SLAcontrol module determines if the requested traffic class has sufficientbandwidth to support the traffic flow. If the determined class does nothave sufficient bandwidth, the module informs the services manager 120that the admission request is denied. If the determined class hassufficient bandwidth, the module admits the request by reducing theavailable bandwidth for that class by the amount requested and returnsto the services manager the DSCP value associated with the trafficclass.

Turning to the middleware module 122, FIG. 2 shows a high levelfunctional architecture of the module (Note that for descriptionpurposes, the middleware module is shown in FIGS. 1-7 as a single entityper host with each entity serving multiple application instances.Preferably, each application instance at a host has a correspondingindependent instance of the middleware module). Note that the module 122is identical whether located at a client 102 or a server 104. Notefurther that the while FIG. 2 shows the middleware module being residentat a client and a server that are a communicating, each instance of themodule is independent of the other and need not be resident at both. Themodule need only be resident at any client or server that containsapplications needing to request QoS services from network 106 for thetraffic flows it generates. For example, assume client application 110 aand server application 112 a establish a session 216 comprisingclient-to-server traffic flow 216 a and server-to-client traffic flow216 b. If only the client-to-server flow 216 a requires QoS services,then only the middleware module 122 a is needed. Similarly, if only theserver-to-client flow requires QoS services, then only the middlewaremodule 122 b is needed.

As shown, the middleware module is software-based and comprises amiddleware control process 202, which controls the QoS configurationsfor all traffic flows any application instance on the client or servergenerates, a middleware API (application programming interface) module204, which the applications use to request QoS services from system 100,and a middleware QoS configuration module 206, which performs QoSconfigurations on the client or server for the traffic flows. The API islinked with the client and server applications 110 a/b and 112 a/brespectively and as such and in accordance with this first embodiment,application source code must be modified to interact with system 100.

In particular, once an instance of a client application and serverapplication establish an application session, the client and/or serverapplication uses the middleware API 204 to make QoS requests for thetraffic flows it generates over the session. As indicated, anapplication may choose to make only one QoS request, indicating that theQoS services should last for the life of the session, or may makenumerous QoS requests, each request coming prior to thesending/receiving of a particular traffic flow and indicating that therequested QoS services should be short-lived. Similarly, an applicationcan terminate priorly requested QoS services.

When an application instance requests QoS services for a givensession/traffic flow, it specifies the client's and server's IPaddresses, the port numbers of the session, and an application ID/flowID, which identify the type of application and flow. The applicationinstance may also specify high-level traffic flow characteristics.Specifically, the application can specify the duration of the flow, theflow size, the bandwidth required, and a service class, which is a highlevel description of the type of data transfer (i.e., a bulk datatransfer, a video transfer, etc). However, a disadvantage of theapplication characterizing the flow is that the description is nowhard-coded in the application and not easily modified. Accordingly, theapplication may alternatively specify no description or only a partialdescription of the traffic flow. For example, the application may onlyspecify the flow duration and flow size leaving the bandwidth andservice class unspecified. In this case and as further described below,when the services manager receives the request, it maps the applicationand traffic flow IDs to a default set of traffic flow characteristicsthat it maintains in a database, thereby providing the remaining values.The advantage of this latter method is that an application developerneed not statically program the flow characteristics in the application.If at a later time it is determined that different trafficcharacteristics should be used for the traffic flow, an administratorneed only update the database of default descriptions at the servicesmanager.

Once the middleware API 204 is invoked to make a QoS request, themiddleware control process 202 receives the request. This process isinitiated prior to a given application instance requesting QoS servicesfrom system 100. When initiated, it establishes a connection 208/230,such as a TCP socket connection, to the services manager 120 and usesthis connection to convey the QoS requests to the services manager onbehalf of the application instances 110 a/b and 112 a/b requesting QoSservices. Accordingly, upon receiving a QoS request from an application,the middleware process determines the user-ID/login-ID of theapplication user, if available, and then conveys the request and user-IDto the services manager 120 through connection 208/230.

Once the services manager receives a QoS request from an applicationinstance via the middleware control process 202, it determines ifnetwork 106 can support the QoS request for the specified traffic flowand either denies the request or obtains a DSCP value when the requestis granted. When the request is denied, the services manager informs thecontrol process 202, which in turn informs the calling applicationinstance. When the request is granted, the services manager 120 returnsthe DSCP value to the middleware control process 202, which in turninvokes the QoS configuration module 206 to perform QoS configurationsat the client 102 a or server 104 a for the application instance.

Specifically, upon being invoked by the middleware control process, theQoS configuration module 206 issues I/O (input/output) control messagesto the local operating system 218/228 (specifically, to the TCP/IPstack) instructing the operating system to configure the session for theapplication instance such that the DSCP field of all transmitted packetson the session are marked with the DSCP value. The control process 202then notifies the calling application instance that the QoSconfiguration is complete. Accordingly, rather than reconfiguring theedge routers and burdening them with the packet-marking function, system100 distributes this function to the traffic sources.

Once network QoS resources are allocated to a given traffic flow, theresources are released either explicitly through an application-issuedrequest using the middleware API module 204 or automatically through theexpiration of the duration specified in the QoS request. When anapplication instance initiates the release, the middleware controlprocess 202 receives the request and conveys the request throughconnection 208/230 to the services manager 120, which reallocates thenetwork resources. The services manager then confirms the release withthe control process 202, which in turn uses the QoS configuration module206 to issue an I/O control message to the local operating system218/228 instructing the operating system to configure the session forthe application instance such that the DSCP field of all outgoingpackets is no longer marked.

Regardless of whether a release request is issued, the services manager120 also automatically monitors the length of allocated QoS resources toa given flow based on the duration specified in the QoS request. At theend of the specified duration, if the application has not yet releasedthe resources, the services manager automatically reallocates theallocated QoS resources and then proceeds as just described, notifyingthe middleware control process 202.

Turning to the services manager 120, it is a software-based system thatacts as a central entity that oversees QoS management for a set ofclients 102 and servers 104 communicating over network 106. FIG. 3 showsa functional architecture of the services manager, which comprises aservices manager control process 302, a middleware communication module304, a network configuration control module 306, an SLA control modules320, a QoS manager 308, a policy manager 310, a topology manager 312,and a set of databases including a default QoS database 314, a networktopology database 316, and an alternate QoS database 318.

The middleware communication module 304 manages all connections 208 and230 between the services manager and the middleware control processes202 at the clients 102 and servers 104, and in particular, passes QoSrequests/responses between these processes and the services managercontrol process 302.

The services manager control process 302 oversees the processing of theapplication initiated QoS requests. Upon receiving a QoS request from anapplication, the services manager control process first interacts withthe policy manager 310 to obtain a complete characterization of thetraffic flow. As indicated above, rather than fully specifying thetraffic flow characterization when making a QoS request, an applicationcan leave it partially or totally unspecified, providing only anapplication ID and flow ID. Accordingly, the services manager controlprocess interacts with the policy manager, which for a given applicationID and flow ID maintains default traffic flow characterizations in thedefault QoS database 314. Again, by maintaining defaultcharacterizations at the services manager, an administrator can updatethe characterization for a given application and flow as needed, ratherthan hard-coding the characterization in the application. Moreimportantly, an administrator can establish a set of policies that thepolicy manager uses to determine a default characterization for a givenapplication ID and flow ID.

In addition to maintaining default characterizations, the default QoSdatabase 314 can also maintain traffic flow characterizations forindividual users and indexed by user-ID/logon. This allows users toindividually customize the QoS that will be given to the traffic flowsthey each generate for a given application. Accordingly, if auser-ID/login is specified in the QoS request, the policy manager 310also makes this determination when finalizing the traffic flowcharacterization for a given flow. Once having a complete traffic flowcharacterization, the services manager control process converts thischaracterization into a detailed set of low-level traffic flowattributes, uses the flow ID to determine the direction of the flow, andthen passes the QoS request to the QoS manager for admission control.

The QoS manager 308 oversees the actual QoS admission to network 106 anddetermination of a DSCP value for a given traffic flow. In particular,upon receiving a QoS request from the services manager control process302, the QoS manager first contacts the topology manager 312 todetermine the sub-networks 106 traversed by the specified traffic flow,as indicated through the client and server's IP addresses specified inthe QoS request. The topology manager maintains the network topologydatabase 316, which includes a list of entries wherein the index to eachentry is a source/destination host-IP address pair (or subnet addresspair) and wherein each entry is a list of network addressescorresponding to the administrative domains over which traffic betweenthe source and destination IP address is carried. An administrator canmanually maintain the topology database 316 or preferably, the topologymanager 312 can dynamically maintain the database. For example, thetopology manager can periodically run the trace-route program for hostor subnet address pairs and use the results to populate the database.

The QoS manager 308 provides the topology manager with the client andserver IP addresses for the specified traffic flow and in responsereceives the network addresses of the sub-networks 106 through which thegiven traffic flow will move. The QoS manager then determines whichsub-networks are Diff Serv enabled and in particular, determines thenetwork configuration system 124 and/or SLA control module 320 for eachof these networks. For each managed network, the QoS manager formulatesa QoS admission request specifying the low-level traffic flowattributes, the direction of the traffic flow, and the client/server IPaddresses. The QoS manager then issues each request to the appropriatenetwork configuration system 124 and/or SLA control modules 320. Withrespect to the network configuration systems 124, these requests gothrough network configuration control module 306, which maps the trafficflow attributes to one of the traffic classes supported by eachsub-network and then issues admission requests for these traffic classesto the network configuration systems 124. With respect to the SLAcontrol modules 320, again, each of these modules converts the trafficflow attributes to a specific traffic class supported by itscorresponding network and then performs admission control as describedabove.

Assuming any of the network configuration systems 124 and/or SLA controlmodules 320 receiving a QoS request from the QoS manager cannot admitthe request, the QoS manager returns control to the services managercontrol process 302, informing it that the admission has been denied.When this occurs, the services manager control process interacts withthe policy manager 310 to obtain an alternative characterization for thetraffic flow. In particular, in addition to maintaining defaultcharacterizations, the policy manager also maintains alternativecharacterizations through the alternate QoS database 318, which isindexed by application ID and flow ID. If the policy manager determinesan alternative set of characterizations has been defined, the servicesmanager control process 302 maps one of these to low-level traffic flowattributes and passes these new attributes to the QoS manager 308, whichagain attempts to admit the request (with the intent that the trafficflow will now be mapped to traffic classes that have availablebandwidth). Note that the policy manager may maintain multiple sets ofalternate characterizations for a given application ID and flow ID,allowing for multiple admission attempts. If all admission attemptsfail, the services manager control module notifies the callingapplication as described above.

However, assuming each network configuration system 124 and/or SLAcontrol modules 320 can admit the QoS request, each returns a DSCP valuecorresponding to a determined traffic class to the QoS manager. The QoSmanager returns to the services manager control process the DSCP valueof the first Diff Serv enabled sub-network the traffic flow traverses,thereby indicating the QoS request has been admitted. The servicesmanager control process 302 then forwards the DSCP value via themiddleware communication module 304 to the middleware control process202 on the client or server, depending on which made the request.

Note that as described above, adjacent sub-networks are presumed to bepre-configured to automatically perform packet remarking between thedifferent traffic classes. However, system 100 can also configure thenetworks to perform this remarking. For example, rather than containinga list of network addresses for each host-IP address pair, the networktopology database 316 can comprise a list of entries wherein the indexto each entry is still a source/destination host-IP address pair butwherein each entry is a list of sub-network address pairs showing theingress and egress IP addresses of each sub-network traversed in orderto connect the corresponding host-IP address pair. Using thisinformation and the DSCP values returned by the network configurationsystems 124 and/or SLA control modules 320, the QoS manager can instructthe appropriate network configuration systems to perform theseremarkings.

As indicated, the resources allocated to a given traffic flow arereleased either explicitly through an application-issued request orautomatically through the expiration of the duration specified in theQoS request. When an application issues a release, the release isreceived by the services control module 302, which forwards it to theQoS module 308, which in turn issues a release to each of the networkconfiguration systems 124 and/or SLA control modules 320 over-seeing thesub-networks 106 through which the traffic flows. Regarding theexpiration of a specified duration, the services manager control modulemaintains timers for each admitted QoS request and automaticallyinstructs the QoS module to release allocated QoS service upon theexpiration of the timer.

As described above, system 100 allows a client or server application tomake QoS requests for the traffic flows that client or serverapplication generates. FIG. 4 is a high level illustrative diagram of asecond embodiment of system 400 wherein client and server applicationscan continue to make QoS provisioning requests for the traffic flowseach generates but can also make QoS provisioning requests for thetraffic flows each receives. Similar to the first embodiment, system 400comprises a middleware module 122 at clients and/or servers 402-406, aservices manager 422, and network control systems 124. Each of thesemodules continues to function as described above with the servicesmanager performing additional functions as described below. In addition,system 400 also comprises a policy enforcement module 420 at clientsand/or servers, which module 420 performs DSCP marking configurations atthe client/server for the traffic flows the applications at thisclient/server generate but wherein the module is activated by anopposing host (Note that unlike the middleware module, only a singleinstance of the policy enforcement module exists at a given host andserves multiple applications). The exact configuration of which clientsand servers contain middleware modules 122 and policy enforcementmodules 420 is not specific to this embodiment. As such, some clientsand servers may contain a middleware module, a policy enforcementmodule, or both. The determination is essentially based on theapplications any given host is running. It should also be noted that thepolicy enforcement module and each instance of a middleware module isindependent of the other modules, even when the modules are running onthe same host.

As an example of this second embodiment, consider client 402 and server404. Here, client 402 comprises a middleware module 122 and server 404comprises a policy enforcement module 420. Assume that application 410at client 402 and application 412 at server 404 have an applicationsession with client-to-server and server-to-client flows. Application410 can make a QoS provisioning request for the client-to-server flow,with the middleware module 122 relaying the request to the servicesmanager 422 and with the middleware module also configuring the client402 with the obtained DSCP value for packet marking, as described above.However, application 410 can also make QoS provisioning requests for theserver-to-client flow. Here, application 410 again makes a QoSprovisioning request, which the middleware module 122 relaying therequest to the services manager 422. Similar to above, the servicesmanager determines traffic attributes for the flow, determines thesub-networks the traffic flow traverses, and makes QoS admissionrequests to the appropriate network control systems 124 and/or SLAcontrol modules. Assuming each sub-network has sufficient bandwidth toadmit the new traffic flow, the services manager obtains a DSCP valuecorresponding to the first sub-network the traffic flow traverses.However, rather than conveying the DSCP value back to the middlewaremodule 122 at client 402, the services manager conveys the DSCP value tothe policy enforcement module 420 at server 404. The policy enforcementmodule 420, upon receiving the DSCP value from the services manager,configures the server 404 to intercept the traffic flow packetsgenerated by application 412 prior to transmission and to appropriatelymark the DSCP field of the packets, completing the overall QoSconfiguration.

As another example, consider client 406 and server 408. Here, both theclient and server comprise the middleware module 122 and the policyenforcement module 420. Again, assume that application 414 at client 406and application 416 at server 408 have an application session withclient-to-server and server-to-client flows. Here, because the clientand server contain both the middleware module and policy enforcementmodule, application 414 can make QoS requests for the client-to-serverflow and for the server-to-client flow and similarly, application 416can make QoS requests for the server-to-client flow and for theclient-to-server flow.

Reference will now be made in greater detail to the policy enforcementmodule 420, with FIG. 5 showing a functional design of the module. Notethat for discussion purposes, FIG. 5 uses the first example from abovewith client 402 comprising the middleware module 122 and with server 404comprising the policy enforcement module 420. As shown, the policyenforcement module 420 is a software-based module and comprises a policyenforcement control process 502, a policy enforcement QoS configurationmodule 504, and one or more traffic filters 506. Similar to themiddleware module 122, the policy enforcement module is initiated priorto any given application instance requesting QoS services from system400.

Once a client application (such as application instance 410) establishesa session (such as session 514) to a server application (such asapplication instance 412), the client application uses the middlewareAPI to request QoS services for client-to-server traffic flows (such asflow 514 a) and/or for server-to-client traffic flows (such as flow 514b) occurring over the application session. Again, the applicationspecifies an application ID and a flow ID. As the services managerreceives the request from the middleware module over connection 516, itoperates similar to system 100, and attempts to admit the flow tonetwork 106. Assuming the flow is admitted and the services managerobtains a DSCP value, it determines the direction of the flow based onapplication/flow IDs and conveys the value to either the middlewaremodule or policy enforcement module.

In particular, for a server-to-client flow, the services manager nextestablishes a connection 508, such as a TCP socket connection, to thepolicy enforcement control process 502 at the server 404. The servicesmanager then communicates with the policy enforcement control processthrough this connection, passing it the determined DSCP value and theclient/server IP addresses/port numbers of the session 514 (as providedby the middleware control process) thereby instructing the process 502to perform a packet-marking configuration.

In general, the policy enforcement control process 502 performs thepacket-marking configuration by invoking the policy enforcement QOSconfiguration module 504, which configures the local operating system512 to mark the DSCP field of all outgoing packets on theserver-to-client traffic flow 514 b. Specifically, one way the policyenforcement QOS configuration module 504 performs this configuration isto install a traffic filter 506 that resides within the operating systemkernel 512 below the TCP/IP stack 510 and that filters on thesource/destination IP address and port number information of transmittedpackets, capturing all packets corresponding to traffic flow 514 b. Asthe desired packets are captured, the traffic filter marks the DSCPfield with the specified DSCP value and then retransmits the packetsonto network 106. The traffic filter capability can be implemented usingthe Solaris Bandwidth Manager for hosts running the Solaris OperatingSystem or using the Windows XXX for hosts running the Windows OperatingSystem. Note that the services manager similarly communicates with thepolicy enforcement control process 502 to remove the traffic filter wheninstructed through a remove request or when a duration timer expires.

Reference will now be made to a third embodiment where in addition tothe clients and servers performing the packet marking function, theclients and servers also can perform the packet policing and shapingfunctions for the traffic flows they generate. Similar to the firstembodiment, this embodiment comprises a middleware module at the clientsand/or servers, a services manager, and network control systems. Each ofthese modules continues to function as described above with respect tothe first embodiment. In addition, like the second embodiment, thisthird embodiment can also comprise a policy enforcement module at remotehosts that is used by local hosts to make QoS requests for the trafficflows the remote host generates. However, in accordance with this thirdembodiment, this policy enforcement module also performs the packetpolicing and packet shaping functions for the traffic flows the remotehost generates. In addition, in the case where a local host is makingQoS requests for the traffic flows it directly generates, this thirdembodiment adds a policy enforcement module to the local host thatperforms the packet policing and packet shaping functions for thesetraffic flows.

FIG. 6 is an exemplary configuration of the third embodiment thatillustrates the functional operation of packet shaping and packetpolicing. As shown, client 602 comprises a middleware module 122 andserver 604 comprises a policy enforcement module 618 thereby allowing aclient-based application (such as application instance 638) to establisha session (such as application session 642) to a server-basedapplication (such as application instance 640) and to make QoS requestsfor client-to-server traffic flows (such as traffic flow 642 a) and forserver-to-client traffic flows (such as traffic flow 642 b) as describedabove. With respect to the server-to-client flows, the policyenforcement module 618 at server 604 continues to perform the packetmarking function. However, this module now also performs the packetshaping and policing function for these flows. With respect to theclient-to-server flows, the middleware module 122 at the client 602continues to perform the packet marking function. However, for thepacket policing and shaping functions with respect to these flows, apolicy enforcement module 606 is now also needed at the client 602.

More specifically, with respect to the server-to-client flows,application 638 makes a QoS request to the middleware module 122, whichconveys the request to the services manager 636 over connection 630.Once the services manager 636 obtains a DSCP value for the requestedflow, it establishes a connection 634 to the policy enforcement controlprocess 620 at server 604 and passes it the DSCP value and theclient/server IP addresses/port numbers of the session 642 as describedabove. However, it also conveys rate control information for thepolicing and shaping of the traffic flow 642 b. The policy enforcementcontrol process 620 then invokes the policy enforcement QOSconfiguration module 622 to perform the packet marking and rate controlconfiguration for the traffic flow. Again, one way the policyenforcement QOS configuration module 622 can do this is to install atraffic filter 624 that resides within the operating system kernel 626below the TCP/IP stack 628 and that filters on the source/destination IPaddress and port number information of transmitted packets, capturingall packets corresponding to traffic flow 642 b. As the desired packetsare captured, the traffic filter marks the DSCP field with the specifiedDSCP value and then retransmits the packets onto network 106 inaccordance with the specified rate control information.

With respect to client-to-server flows, application 638 makes a QoSrequest to the middleware module 122, which conveys the request to theservices manager 636 over connection 630. Once the services manager 636obtains a DSCP value for the requested flow, it coveys the value to themiddleware control process 202 over connection 630, which in turn usesthe QoS configuration module 206 to configure the local operating system616 to mark the DSCP field of all outgoing packets on the traffic flow642 a, as described above. However, in addition, the services manager636 also establishes an additional connection 632, such as a TCP socketconnection, to the policy enforcement control process 608. The servicesmanager then conveys to the policy enforcement control process ratecontrol information (for policing and shaping) and the client/server IPaddresses/port numbers of the session 642. The policy enforcementcontrol process 608 in turn invokes the policy enforcement QOSconfiguration module 610 to configure the local operating system 616 toperform the rate control functions on the outgoing packets of theclient-to-server traffic flow 642 a. Specifically, one way the policyenforcement control process can do this is to install a traffic filter612 that resides within the operating system kernel 616 below the TCP/IPstack 612 and that filters on the source/destination IP address and portnumber information of transmitted packets, capturing all packetscorresponding to traffic flow 642 a. As the desired packets arecaptured, the traffic filter retransmits the packets onto network 106 inaccordance with the specified rate control information. Again, thetraffic filter capability can be implemented using the Solaris BandwidthManager for clients running the Solaris Operating System or using theWindows XXX for clients running the Windows Operating System.

FIG. 7 is an illustrative diagram of a fourth embodiment of system 700that provisions end-to-end QoS services on a per application flow basis.As described above, under the first three embodiments applications aremodified to include the middleware API 204 in order to directly controlthe QoS requests for a given application session and traffic flows.System 700 continues to support this function (as shown by applicationinstance 704) but, in addition, allows a user to make QoS requests forapplication sessions and traffic flows wherein the application itself,such as application instance 706, is not modified to include themiddleware API. This allows users to make QoS requests for legacyapplications, for example. Note that for discussion purposes, FIG. 7 isonly one exemplary configuration of system 700 for the purpose ofshowing the functional operation of how a user can directly initiate QoSrequests for client-to-server and server-to-client traffic flows.However, it is readily apparent that this embodiment can also becombined with the first three embodiments described above.

System 700 comprises a services manager 730 and one or more networkcontrol systems 124 a-b that operate as described above, overseeing theQoS management for network 106 on behalf of clients and servers. Inaddition, system 700 comprises a signaling client 720 and a middlewaremodule 122 that operate together to allow a user to make the QoSrequests for client-to-server and server-to-client flows. (As indicated,the middleware module can also continue to operate directly withapplications.) Typically, the signaling client/middleware module will belocated at the client side of a communication, which assumption is madein the following discussion and is shown in FIG. 7 for descriptionpurposes. However, again, other configurations are possible. When thesignaling client is used to make QoS requests, the middleware module canno longer perform the packet marking function for locally generatedtraffic flows. As such, system 700 also comprises the policy enforcementmodule 722 at both local and remote hosts, which module performs thepacket marking (and policing/shaping functions) whenever the signalingclient 720 is used to initiate QoS requests. Hence, a policy enforcementmodule 722 is essentially located at any client and server generatingtraffic flows for which a user wants to make QoS requests. Lastly, theservices manager 730 of system 700 comprises an application instancedatabase 738 and a modified services manager control process 732, amodified middleware communication module 734, and a modified policymanager 736 as compared to the first three embodiments. (The remainingelements of services manager 730 are the same as the services manager asdescribed in reference to FIG. 3 and are not shown in FIG. 7 forsimplicity.)

In accordance with system 700, a user uses signaling client 720 to makeQoS requests for application sessions and traffic flows corresponding toapplications that are not modified to include the middleware API 204,such as application instance 706 at client 702. Here, the signalingclient 720, which is independent of the applications, includes themiddleware API and based on user initiation, conveys a QoS request for aparticular application session/traffic flow to the services manager 730via the middleware module 122 and connection 740. If the flow isadmitted, the services manager obtains a DSCP value and determines thedirection of the flow. For server-to-client flows, the services managerconveys the DSCP value to the policy enforcement module 722 at server750, similar to above. However, for client-to-server traffic flows, themiddleware module is not able to perform the necessary QoSconfigurations at the client 702. Accordingly, the services mangerestablishes a new connection 744 to the policy enforcement module 722 atthe client 702 and conveys the DSCP value to this process. The policyenforcement module 722 then operates like the policy enforcement moduledescribed in reference to the second and third embodiments, configuringthe client 702 to mark the packets and possibly perform rate controlfunctions for a client-to-server traffic flow.

More specifically, the signaling client 720 is configured to present toa user (e.g., through a GUI (graphical user interface)) a list ofapplications for which the user can control QoS services. The signalingclient may obtain this list, for example, from a database. Accordingly,once a client and server application have started an applicationsession, a user can select a corresponding application at the signalingclient, indicating that the user wants to make QoS requests for thetraffic flows occurring between the client and server applications(Similarly, the user can designate that the QoS services should bereleased.). Once a user invokes the QoS services, the signaling clientuses the middleware API 204 to make QoS requests (and QoS releaserequests) to the middleware module 122. Like above, these requestsinclude an application ID and a flow ID and preferably exclude thecharacterizations of the traffic flows, allowing the services manager730 to provide these values. However, traffic flow characterizations canalso be hard-coded in the signaling client, could be obtained from alocal database, or could be specified by the user through a GUI, forexample. Nonetheless, while the signaling client 720 may know one of theIP addresses corresponding to the application session, most likely theclient 702 IP address, unlike the first three embodiments the signalingclient does not know the server 704 IP address or the source/destinationport numbers of the application session for which it is requesting QoSservices. This is because the signaling client is running separate fromthe applications. Accordingly, the signaling client leaves unspecifiedthe unknown IP addresses and port numbers. However, as a further option,the user could specify the server host name to the signaling client,which the signaling can convert to an IP address.

Recall that the client/server IP addresses and port numbers areimportant for two reasons. First, the services manager 730 needs the IPaddresses to determine the sub-networks the traffic flow between theclient and server will traverse. Second, for server-to-client trafficflows, the services manager needs to know the server so that it canestablish a connection 742 to the policy enforcement module 722 at theserver 750, for example, in order to instruct the module to configure anappropriate traffic filter to mark the outgoing traffic flow packets.

Once the signaling client makes a QoS request, the middleware controlprocess 202 passes the request, and a user ID, to the services manager730 via connection 740 where the middleware communications module 734receives the request and passes it to the services manager controlprocess 732. Again, the control process interacts with the policymanager 736 to obtain a default and/or user-specific traffic flowcharacterization as described above. In addition, if the user does notspecify the server host name, the control process also interacts withthe policy manager 736 to determine the server IP address and possiblythe port number of the session (for well-known port numbers). Inparticular, in accordance with system 700, the policy manager maintainsan applications instance database 738, which contains a default set ofIP addresses and well-known port numbers indexed by application ID andflow ID (again, with reference to FIG. 7, the database would comprisedefault server IP addresses). Again, this is necessary because unlikethe above embodiments, the signaling client cannot specify the completeendpoints of the session.

At this point, the services manager control process 732 has a completedescription of the QoS request (i.e., traffic flow characterization andthe endpoints of the session) and proceeds as above, passing the QoSrequest to the QoS manager 308. Assuming the QoS request is admitted,for server-to-client traffic flows the control process 732 establishesthe connection 742 to the policy enforcement module 722 via themiddleware communication module 734 and passes the module the DSCP value(and possibly rate configuration information). Note that because atleast the client side port number of the traffic flow is unknown, thetraffic filter the policy enforcement module installs is broader thanunder the prior embodiments.

For client-to-server flows, the control process 732 proceeds similar toserver-to-client flows. Again, system 700 does not know the client sideport number of the session. As such, unlike the prior embodiments themiddleware control process 202 cannot determine the session at theclient 702 and therefore cannot invoke the QoS configuration module 206to configure the local operating system 708 to configure the session tomark the outgoing packets with the DSCP value. Accordingly, the controlprocess 732 establishes an additional connection 744 to the policyenforcement control process 722 via the middleware communication module734 and passes the process the DSCP value, the client and server IPaddresses, possibly the server port number, and possibly rateconfiguration information.

Similar to above, the policy enforcement control process 724 performsthe packet-marking/rate configuration by invoking the policy enforcementQOS configuration module 726, which configures the local operatingsystem 708 to mark the DSCP field all outgoing packets on theclient-to-server traffic flow. Again, one way the policy enforcementcontrol process does this is to install a traffic filter 728 thatresides within the operating system kernel 708 below the IP stack 710and that captures all packets on the flow.

The above-described embodiments are intended to be illustrative only.Numerous other embodiments may be devised by those skilled in the artwithout departing from the spirit and scope of the invention.

1. A services manager for managing quality of service (QoS) for trafficflows generated by a plurality of hosts interconnected by one or morenetworks, wherein at least one of the networks is enabled with a set oftraffic classes, the services manager comprising: means for receiving aQoS provisioning request from a middleware module for any given trafficflow; means for determining traffic attributes for the given trafficflow using a topology database, wherein the traffic attributes includewhich of the one or more networks the given traffic flow traverses;means for obtaining a DSCP (Differentiated Services Code Point) valuefor the given traffic flow based on whether the one or more networks thetraffic flow traverses is configured to support the flow given thedetermined traffic attributes; and means for conveying the obtained DSCPvalue to a first host when the traffic flow is from the first host to asecond host.
 2. The services manager of claim 1, wherein the conveyingmeans further conveys the obtained DSCP value to the second host whenthe traffic flow is from the second to the first host.
 3. The servicesmanager of claim 1, wherein the obtained DSCP value corresponds to adetermined network.
 4. The services manager of claim 1, wherein the DSCPobtaining means comprises a network control module for interfacing oneor more network control systems, wherein each network control systemcorresponds to a network that has an enabled set of traffic classes, andwherein each network control system performs admission control for thatnetwork by determining if a given traffic class corresponding to thedetermined traffic attributes has sufficient available bandwidth tosupport the traffic flow and, if so, returns a DSCP value for thattraffic class.
 5. The services manager of claim 4, wherein the DSCPobtaining means further comprises one or more SLA (service levelagreement) control modules, wherein each SLA control module correspondsto a network to which the plurality of hosts have a service levelagreement for an enabled set of traffic classes, and wherein each SLAcontrol module performs admission control over the service levelagreement for its corresponding network by determining if a giventraffic class within the service level agreement and corresponding tothe determined traffic attributes has sufficient available bandwidth tosupport the traffic flow and, if so, returns a DSCP value for thattraffic class.
 6. The services manager of claim 1, wherein the trafficattribute determining means comprises a default QoS database with aplurality of default traffic flow characterizations, and wherein thetraffic attribute determining means uses the database to determine adefault traffic flow characterization for the given traffic flow andconverts the determined default traffic flow characterization to thetraffic attributes.
 7. The services manager of claim 6, wherein thetraffic attributes determining means further comprises an alternate QoSdatabase with a plurality of alternate traffic flow characterizations,wherein the traffic attribute determining means uses the alternate QoSdatabase to determine an alternate traffic flow characterization andthereby alternate traffic attributes, and wherein if the DSCP obtainingmeans cannot determine a DSCP value given the determined trafficattributes, the obtaining means obtains a DSCP value for the giventraffic flow based on whether the networks the traffic flow traverses isconfigured to support the flow given the determined alternate trafficattributes.
 8. A system at a host for managing quality of service (QoS)for a plurality of traffic flows traversing one or more networks,wherein at least one of the networks is enabled with a set of trafficclasses, the system comprising: a middleware control module forreceiving QoS provisioning requests for the plurality of traffic flowsand for conveying the requests to a services manager intended fordetermining which of the one or more networks any given traffic flow ofthe plurality of traffic flows traverses and for obtaining a DSCP(Differentiated Services Code Point) value that corresponds to adetermined network that is traffic class enabled and is the firstnetwork the any given traffic flow traverses, and wherein the middlewarecontrol module is for receiving the determined DSCP values for one ormore of the traffic flows generated by the host, and means responsive tothe middleware control module for using the DSCP values for the one ormore of the traffic flows generated by the host to mark the transmittedpackets corresponding to these traffic flows.
 9. The system of claim 8,further comprising a policy enforcement module for policing and packetshaping the transmitted packets corresponding to one or more of trafficflows generated by the host.
 10. The system of claim 8, furthercomprising means for conveying the obtained DSCP value to a second hostwhen the traffic flow is from the second to the first host.
 11. Thesystem of claim 8, further comprising: means for determining a defaulttraffic characterization for the given traffic flow based on a flowidentification in the provisioning request, and means for converting thedefault traffic characterization into the traffic attributes.
 12. Thesystem of claim 8, comprising: means for determining an alternatetraffic flow characterization for the given traffic flow, means forconverting the default traffic characterization into alternate trafficattributes, and means for obtaining a DSCP value for the given trafficflow based on whether the one or more networks the traffic flowtraverses is configured to support the flow given the determinedalternate traffic attributes.
 13. A system for managing quality ofservice (QoS) for traffic flows generated by a plurality of hostsseparated by one or more networks, wherein at least one of the networksis enabled with a set of traffic classes, the system comprising: aservices manager, and a middleware module at least one of the pluralityof hosts, wherein the middleware module at the one host receives a QoSrequest for a traffic flow the host generates and conveys a QoSprovisioning request to the services manager upon receiving the QoSrequest for the traffic flow, and wherein the services manager receivesthe QoS provisioning request from the middleware module, obtains a DSCP(Differentiated Services Code Point) value for the traffic flow only ifit is determined that the one or more networks the traffic flowtraverses is configured to support the flow, and if the DSCP value isobtained, conveys the obtained DSCP value for the traffic flow to themiddleware module, and wherein the middleware module uses the obtainedDSCP value received from the services manager to mark the DSCP field ofpackets of the traffic flow; wherein the services manager, uponreceiving the QoS provisioning request, determines the one or morenetworks the traffic flow traverses, and as part of obtaining the DSCPvalue further determines if for each traffic class enabled network thetraffic flow traverses there is sufficient bandwidth in a traffic classto support the traffic flow; and wherein the QoS request contains anidentification of the traffic flow and wherein the middleware moduleconveys this identification to the services manager as part of the QoSprovisioning request.
 14. The system of claim 13, wherein the servicesmanager, upon receiving the QoS provisioning request, determines adefault traffic flow characterization for the traffic flow based on thetraffic flow identification, and uses the default traffic flowcharacterization to obtain a DSCP value by determining if the one ormore networks the traffic flow traverses is configured to support theflow based on the characterization.
 15. The system of claim 13, whereinif the services manager cannot obtain a DSCP value based on thedetermined default traffic flow characterization, the services managerdetermines an alternate traffic flow characterization for the trafficflow based on the traffic flow identification, and uses the determinedalternate traffic flow characterization to obtain a DSCP value bydetermining if the one or more networks the traffic flow traverses isconfigured to support the flow based on the alternate characterization.16. The system of claim 13, further comprising a policy enforcementmodule at least one of the plurality of hosts, wherein the one hostcontaining the policy enforcement module generates a second trafficflow, and wherein the middleware module receives a QoS request for thesecond traffic flow and conveys a QoS provisioning request to theservices manager upon receiving the QoS request for the second trafficflow.
 17. The system of claim 16, wherein the services manager obtains aDSCP value for the second traffic flow if the networks the secondtraffic flow traverses can support the flow, and if the DSCP value forthe second traffic flow is obtained, conveys the obtained DSCP value forthe second traffic flow to the policy enforcement module.
 18. The systemof claim 16, wherein the policy enforcement module uses the obtainedDSCP value received from the services manager to mark the DSCP field ofpackets of the second traffic flow.
 19. The system of claim 16, whereinthe services manager further conveys, when a DSCP value is obtained forthe second traffic flow, packet policing and packet shaping instructionsto the policy enforcement module, and wherein the policy enforcementmodule uses the packet policing and packet shaping instructions receivedfrom the services manager to police and shape the packets of the secondtraffic flow.
 20. The system of claim 13, further comprising a policyenforcement module at the one host, and wherein when the DSCP value isobtained for the traffic flow, the services manager further conveyspacket policing and packet shaping instructions to the policyenforcement module, and the policy enforcement module uses the packetpolicing and packet shaping instructions received from the servicesmanager to police and shape the packets of the traffic flow.